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ABSTRACT 

Intended to help the Hawaii Vocational Rehabilitation 
and Services for the Blind administration improve productivity, this 
report focuses on the measurement of productivity and on the use of 
program information to monitor performance and to investigate the 
causes of productivity problems. A process is recommended for 
establishing performance objectives on agency goals, using a phasing 
approach where performance objectives are set in the short-term for 
top priority goals. A recommendation is then made for short-term 
prioritization of agency goals. These indicators are suggested for 
top priority: the number of clients closed as rehabilitated, the 
proportion of clients closed as rehabilitated out of all clients who 
are accepted for service, the proportion of clients closed into 
competitive employment, the proportion of noncompeti t ively employed 
rehabilitated clients showing improvement in life status and 
functional abilities, total costs per rehabilitation, and percent ot 
severely handicapped clients accepted for service. As an introduction 
to the use of performance measures for management decision making, an 
example is presented of the use of productivity information to 
identify administrative actions that should be taken when indicators 
point to problematic performance. ( Author/YLB ) 
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ABSTRACT 



RECOMMENDATIONS FOR ESTABLISHING PERFORMANCE OBJECTIVES ON PRODUCTIVITY 
GOALS BY HAWAII VOCATIONAL REHABILITATION AND SERVICES FOR THE BLIND 
(HVRSB) 

Report by J. Mark Rogers, Berkeley Planning Associates, February 1983, 
35 pp. 

The HVRSB administration has expressed an interest in improving its 
productivity. This report focuses on the measurement of productivity and 
on the use of program information to monitor performance and investigate 
the causes of productivity problems. The report includes a recommended 
process for establishing performance objectives on agency goals, using 
a phasing approach where performance objectives are set in the short term 
for top priority goals. Indicators suggested for top priority include: 
the number of clients closed rehabilitated; the proportion of clients 
closed rehabilitated out of all clients who are accepted for service; the 
proportion of 26 closures closed into competitive employment; the pro- 
portion of noncompetitively employed 26 closures showing improvement in 
life status and functional abilities; total costs per rehabilitation; and 
percent of severely handicapped clients accepted for services. Finally, 
as an introduction to the use of performance measures for management 
decision -making, the report presents an example of use of productivity 
information to identify administrative actions which should be taken 
when indicators point to problematic performance. 

Agency Contact: Toshio Nishioka 

Hawaii Vocational Rehabilitation and Services 

for the Blind 
Department of Social Services 
P. 0. Box 339 
Honolulu, HI 96809 
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I . INTRODUCTION 



The Rehabilitative Services Administration (RSA) is sponsoring a three- 
year project to provide management consulting and technical assistance tc 
state Vocational Rehabilitation agencies, to assist those agencies in devel- 
oping effective planning, management, and information systems. As part of 
this activity, a TA consultant from Berkeley Planning Associates (BPA) con- 
ducted a site visit to the Hawaii Vocational Rehabilitation and Services 
for the Blind (HVRSB) . Prior to the visit, the consultant conducted tele- 
phone discussions with various HVRSB personnel, and received and reviewed 
several documents pertaining to the agency's structure, past performance, 
and management information system. 

The HVRSB administration expressed an interest in improving its pro- 
ductivity. This report focuses on the measurement of productivity and on 
the use of program information to monitor performance and investigate the 
causes of productivity problems. From our discussions on-site with agency 
staff, it is clear that central office management has been seeking ways to 
expand its evaluation of productivity. At the same time, the agency adminis 
tration has wisely recognized the need to avoid imposing too many new pro- 
ductivity goals on staff too quickly. This is especially important given 
that, with the goals put forth by HVRSB management, improvement on one goal 
might mean decreased performance on other goals. In an attempt to overcome 
these conflicts while still maintaining an ongoing thrust toward improved 
overall productivity, BPA has developed the following for HVRSB: 

• a recommended process for establishing performance objectives 
on agency goals (Section II of this report). Our recommendation 
revolves around a phasing approach, where performance objectives 
are set in the short-term only for "top priority" goals. Over 
the short-term period, program analysis is conducted to understand 
the type and extent of tradeoffs existing between the various 
goals, and to understand the ways in which those tradeoffs can 
be overcome. With findings from these analyses, the agency is 
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then in a position to set performance objectives across the 
full range of goals (i.e., M top priority 11 and "other 11 goals) 
in such a way that overall productivity is maximized. 

• A recommendation for short-term prioritizat ion of agency goals 
(Section III of this report). Here we present our suggestions 
as to the indicators which we feel will provide the greatest 
impact on overall productivity, and therefore which we. feel 
should be emphasized in the short-term. 

The agency's Staff Services Office f$3G) will play a key role in 
ensuring the link between management information and operational decision- 
making in the field. SSO will be responsible not only for responding 
to field requests for information, but also for assisting field staff in 
interpreting performance and operational data. In support of this function, 
we have developed: 

• an example of the use of program information in managing VR 
operations (Section IV of this report) . This section discusses 
hypothetical examples of the process SSO might follow in analyzing 
program information for purposes of identifying the causes of 
productivity problems. This process embodies the analytic linkage 
between information on productivity or the lack of it, and manage- 
ment interventions aimed at improving productivity. 

In the chapters which follow, we place the issues within the context 
of the general VR program (i.e., Section 110). However, the processes we 
describe are also relevant to other sections of HVRSB's total scope of 
operations. Regardless of the particular activity, HVRSB will still want 
to identify and prioritize productivity goals and perform analysis of the 
reasons why performance has not met goals. 
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II. A RECOMMENDED PROCESS FOR ESTABLISHING 



PERFORMANCE OBJECTIVES ON AGENCY GOALS 



Concern was expressed by almost all HVRSB staff interviewed that, 
while the agency needed a more complete expression of productivity goals, 
it was important to !l avoid having too many contingency variables 11 in the 
agency's statement of goals. In other words, counselors and line managers 
should not be presented with too many different types of goals, or goals 
that conflict. However, the vocational rehabilitation program has several 
conflicting, though legitimate, performance goals. For example, as outcome 
quality increase (e .g. , wages at closure go up), costs may go up as well 
(since more service and training may be required to achieve greater wages 
at closure). What is needed is a system for setting performance objectives 
that accounts for the tradeoffs among performance goals, consistent with 
the overall policy objectives of the agency. That is, VR agencies must 
understand the con flicts among goals and to set performance objectives . 

While the conflicts among goals are understood at a conceptual level 
(e.g., better outcomes may lead to higher agency costs), the precise trade- 
offs are not well understood (e.g., an "x" percent increase in competitive 
employment closures requires a "y" dollar increase in expenditures per 
rehabilitation). As such, HVRSB cannot establish empirically-based perfor- 
mance objectives which account for specific tradeoffs among goals. However, 
over the next few years, the agency could observe performance across the 
set of goal measures and obtain estimates of the tradeoffs between goals 
(either through "eyebal ling" of performance data, or through more rigorous 
statistical estimation techniques). These estimates could then be used to 
set actual performance objectives. Using this approach, the agency could 
make explicit linkages between its own priorities and known tradeoffs. A 
hypothetical example will illustrate how this would work. 

Assume that HVRSB wishes to increase competitive employment closures 
from a current figure of 50% (hypothetical) to 80%, and that it is known 
from statistical testing "that each 10% increase in competitive employment 
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closures requires a corresponding increase of $50 per rehabilitation. 
Further, assume the agency currently spends $500 per rehabilitation (total 
expenditures, not case service costs). Given this set of circumstances, 
the agency could reasonably expect that a thirty percentage point increase 
in competitive employment closures (from 50% to 80%) would generate a 
$150 increase in total expenditures per rehabilitation (from $500 to $650 
per 26). As such, it would be appropriate, at least for the first year, 
to raise the performance goal for costs per rehabilitation from $500 to 
$650. This assumes that the agency is willing to absorb a $150 increase 
in costs per rehabilitation. 

Alternatively, and consistent with the agency's overall thrust 
towards improved productivity, top management might set goals of 80% com- 
petitive employment and $575-$600 per rehabilitation. In this scenario, 
the agency clearly establishes its intention to achieve a particular goal 
(80% competitive closures) while allowing for a level of "slippage 11 in 
costs per rehabilitation which is less than what would be expected given 
known relationships between outcomes and costs (i.e., a $75-$100 increase 
versus a $150 increase). 

Finally, a third alternative would be for the agency to perform program 
analysis to determine if the desired 80% competitive employment could be 
reached while maintaining the $500/rehabilitation cost. Using this approach, 
the agency would look at its performance data to answer questiors such as: 

• Are there particular service approaches (perhaps used by 
•particular counselors) which lead to the desired proportion 

of competitive employment closures at a lower cost? 
Can these approaches be replicated by other counselors? 

• Are there particular types of clients which cost less to 
rehabilitate into competitive employment? Can the agency 
focus its admissions on these clients while still maintaining 
sufficient admissions of other priority clients and/or 
clients with particular disabilities? 

• Are particular competitive employment job goals easier and 
less costly to obtain (in terms of service costs) and can 
the agency direct clients toward those job goals while still 
being responsive to client wishes? 
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This third alternative would be the preferable approach to setting 
revised numerical performance objectives because it provides a theoretical 
and empirical basis for establishing performance objectives. Thus, based 
on program analysis, top management is able to: 

• determine procedural or policy changes which promise to 
bring desired improvements in productivity; 

• disseminate information to field staff and managers regarding 
the improved "technologies 11 (admissions practices, service 
planning and provision, etc.) ; 

• monitor staff practices in relation to suggested/required 
technologies; 

• intervene in cases of nonjustified noncompliance with new 
policies or procedures; 

• evaluate new policies and procedures with respect to their 
effectiveness in bringing about improved productivity; and 

• validate the precise level of tradeoffs suggested by the 
program analysis. 

While this is the preferable approach to setting performance objectives 
it is of necessity a long-term solution to the agency's needs. In the short 
term, HVRSB needs to establish a goal prioritization through some other 
process. Further, the approach taken must avoid sending too many "signals" 
to field staff in terms of what is expected of them, but must be flexible 
enough to allow modification and addition of goals as more is learned about 
the* tradeoffs involved in meeting conflicting goals. In response to these 
criteria, BPA recommends that the agency adopt a phasing approach to 
establishing productivity goals. This approach uses two phases. Phase I 
includes the following activities: 

• disseminate information on agency goals to staff, via an 
agency policy statement; 

• identify "top priority" goals and secondary goals; 

• establish actual numerical performance objectives for counselors, 
field offices, and other substate units as deemed appropriate 

by management, for the top priority goals ; 
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• on the regular reporting cycle (e.g., monthly or quarterly), 
distribute performance reports to all staff as appropriate. 
For example, each counselor should receive a report 



which shows: current performance on each productivity measure; 
current performance relative to past performance (e.g., previous 
month or year); current performance relative to a higher service 
unit (e.g., counselor's performance compared to the average 
performance for counselors in the field office or the general 
VR services division); and, for top priority goals, performance 
relative to the established performance objective. 
• Over the short-term (two-three years), focus management activi- 
ties on top priority goals . As performance reports are received, 
field managers and counselors should pay particular attention to 
the top priority performance measures for which performance ob- 
jectives have been established. These reports can be used to 
compare a counselors performance against other counselors and 
the field office as a whole, and against the counselor's perfor- 
mance objectives. If particular counselors fail to meet their 
objectives (or if it appears, based on interim reports, that 
some counselors will fail to meet annual objectives), the field 
manager might want to speak with the counselors to get their 
sense of "what went wrong. 11 For example, if a counselor is 
failing to get enough competitive employment closures, the field 
manager might want answers to such questions as: "Are greater 
numbers of such closures imminent? 11 (In which case, •'poor 11 per- 
formance may be merely an artifact of the timing of the performance 
report, to be replaced by "acceptable" performance by the point 
of the next performance report.) Are the kinds of clients served 
by the counselor appropriate for competitive employment? (If not, 
the manager must decide either to lower performance expectations 
for this counselor on the competitive employment measure or to 
direct the counselor to accept fewer such applicants in the future.) 
The role of the Staff Services Off ice in this activity would be 
to respond to requests for in formation from field managers. These 
requests mav consist of special analyses of caseload characteristics, 
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service expenditures, impact of services, and so on. 

• In addition to fielding information requests from district offices, 
SSO would, in this short-term period, undertake data analyses to 
help understand the ways in which field staff can improve pro- 



practices. These analyses would be similar, in terms of content, 
to those conducted at the request of field managers. However, 
they should be broader in the sense that the "unit of analysis 11 
could be the whole agency instead of a counselor or fi3ld office; 
and in the sense that they would be conducted specifically to 
inform major procedural and policy changes of a longer-term nature, 
rather than short-term management-oriented decisions about 
specific counselors or clients. For example, SSO might under- 
take a study of the relative cost-effectiveness of different 
services and service packages in achieving competitive employment 
closures. 

To take a hypothetical situation, SSO might find that, though 
it costs somewhat more to provide job search assistance, clients 1 
chances for obtaining competitive employment are enhanced by 
the service to the extent that the increased cost is deemed 
"worth it. 11 If such were the case, the agency might want to 
mandate (through written policy guidelines) that job search 
"assistance be provided to all clients seeking competitive employ- 
ment, and that cases where it is not provided be documented as 
to the reasons why it was not provided. Other analyses might 
focus on particular types of clients (in terms of disability): 
What types of jobs are these clients successful in obtaining? 
Are counselors with specific types of training more successful 
in working with these clients toward competitive employment? 
Are clients receiving post-employment services substantially 
more likely to retain their jobs? These are but a very few 
of the questions SSO might address in the short term to help 
field staff to refine their techniques and target services 



ductivity on top priority goals through alterations in casework 
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in the most appropriate manner given client circumstances 
and goals, and given available agency resources. 

The management focus on top priority goals during the short term cannot 
be underemphasized, and should be obvious given the above discussion; 
explicit performance objectives are made on these goals, management inter- 
ventions key in on these goals, and SSO f s analytic activities are aimed 
at improving productivity in terms of these goals. Ideally, as a result 
of these various activiti^, performance on top priority productivity goals 
will improve as a result of both increased attention paid to those goals 
(through written policy statements, explicit performance objectives, and 
targeted management review and intervention), and increased understanding 
of the "inputs" (clients, services, counselor training, etc.) which enhance 
the probability of achieving those goals. 

Further, as the agency obtains greater understanding of the techniques 
and procedures which are most effective in achieving top priority goals, 
the agency could increase performance objectives expected of field staff. 
To 1he extent that performance objectives are met for these goals, agency 
management can move ahead to specify performance objectives on the secondary 
goals. If performance objectives on top priority goals have not been 
attained, it would be unwise to further complicate the signals sent 
to field staff through additional performance expectations. Instead, 
the agency should continue to focus its management energies toward those 
goals already identified as top priority. (In fact, the agency might 
even consider focusing on a smaller set of goals for more intensive 
management attention.) However, assuming that performance on top 
priority goals does improve to the point desired by management, then it 
is appropriate to move into Phase II: institutionalizing performance 
objectives on the secondary productivity measures"; — 

"Institutionalization 1 !, will actually have begun long before the end 
of the short-term management' period. Throughout this period, performance 
reports to field managers and counselors will have included performance 
figures on the secondary productivity measures. Though there will hot have 
been specific performance objectives on these measures, and though the 
goals represented by these performance measures will not have been a prin- 
cipal focus of field management interventions, the simple step of providing 
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information on performance will serve to initiate the process of institution- 



report goes out to staff, the agency will have established (e.g., through 
a written memo to all staff) that, in addition to the "top priority" goals, 
there are other secondary goals (with attendant performance measures) on which 
staff will be held accountable at a later time. This policy statement will be 
further reinforced by the actual performance figures received by staff in 
the quarterly or monthly reports. Thus, by seeing actual performance 
numbers for themselves and their counterparts, counselors and field office 
managers may become sensitized to the impending evaluation of their perfor- 
mance in terms of these other indicators. Further, this strategy will 
hopefully stimulate field staff to think about the tradeoffs involved in 
satisfying various goals, which in turn could lead to greater understanding 
of the reasons why tradeoffs exist and how to overcome those tradeoffs to 
maximize achievement of all goals. Thus, over the short-term period, to 
the extent that staff meet top priority goals, both managers and staff may 
be able to turn their attention to improving performance on the other goals. 
It is important to emphasize, however, that this would be appropriate only 
to the extent that performance on top priority goals is meeting top manage- 
ment's expectations. 

The Quality Circle activities used by the agency can play a big role 
in encouraging improved productivity on all agency goals. Reports provided 
to staff will give them an objective viewpoint on which to base discussions 
about how best to meet top priority goals, and how to tailor services to 
the needs of clients so as to maximize performance on all agency goals. 
As staff interact and implement ideas coming out of the quality circle, 
they will be able to track the success of their ideas by observing any 

c hanges (hopefully, improvements) in their individual and collective per- 

formance on productivity measures. Thus, performance reports can become 
a tool not only for targeting management interventions, but also for encour- 
aging discussion among staff about how best to achieve agency goals. The 
end result should be a greater understanding and internalization of agency 
goals, thus making all staff "of one mind 11 as to what those goals are. 
-Heart y , intern a lization w cnirt^:cTir-tref^ 

year short-term period, so there will be no surprises when the time comes 
to evaluate staff on other performance goals. 



alizing these additional productivity goals. Before the first performance 
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Thus, during the short-term, institutionalization of secondary goals will 
have already begun on an informal basis through providing performance reports 
and encouraging discussion about how to achieve these gcals. However, a 
much more formal set of activities should occur at the same time, 
for purposes of identifying tradeoffs and specifying numerical performance 
objectives on other goals. Recall that the agency . wishes to set performance * 
objectives which are realistic; that is, which are attainable given known 
tradeoffs, known technologies for overcoming tradeoffs , and total resources. 
It would be SSO's job to help agency managers determine what is "realistic. 11 
BPA recommends that through the short-term period, SSO regularly conduct 
special statistical anal yses of the performance measures representing secondary 
agency goals. For example: 

• factor analyses could be run at regular intervals, which include 
each of the productivity indicators chosen by agency management. 
Based on these analyses, it may be clear that certain of the 
indicators measure n the same thing 11 (i.e., the same aspect of 
performance). In such a case, it may be appropriate for the 
administration to remove certain indicators from performance 
reports. This will serve to simplify the entire reporting and 
staff evaluation process. 

• In conjunction with the causal analyses on top priority measures, 
SSO would also conduct analyses aimed at understanding the factors 
which influence performance on othergoals. Here, the objective 
.is to identify the services and other inputs (e.g., types of 
counselor training) which are most successful in achieving each 
goal . 

• Correlation and regression analyses would then be used to 
nume r i ca My-estimat e tradeoffs among conflicting goals. As we 

noted earlier, this is the stage where SSO attempts to provide 
answers such as: "A 10% increase in competitive employment 
closures will generate a $50 increase in costs per closure, 
other things being equal. 11 This step would then be followed 
by simultaneous equation models to test for the systems impact 
(i.e., the simultaneous changes in performance across all 
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indicators) resulting from variations in client mix, the 
type, amount, and cost of services provided, etc. The 
results of these analyses will allow the agency to estimate 
the set of tradeoffs as a group, and therefore to forecast 
performance on each indicator, given a particular set of 
operational circumstances. 

Once these analyses are conducted, agency management will need to make 
some policy decisions based on what is learned from the simultaneous equation 
models about the effects of different combinations of service technology. 
Possibly, one combination of technologies might present the optimum approach; 
that is, the approach generating the highest performance on all indicators. 
In this case, the performance scores registered for each indicator could 
serve as the minimum performance objective for staff. More likely, no one 
combination of technologies and procedures will provide optimum performance 
across all indicators. For example, one analysis might suggest that service 
"x 11 generates a 15% increase in the proportion of competitively employed clo- 
sures, but reduces the total number of 26s by 30% and raises costs per com- 
petitively employed closure by 18%. 

Other tests may show different levels and directions of effect on the 
group of indicators. In this/case, policy decisions must be made regarding 
the emphasis to be afforded each goal. For example, should the agency follow 
an approach which maximizes the number of 26s obtained but which minimizes 
the proportion closed competitively and the proportion retaining their jobs 
over time? Alternatively, should the agency try to maximize closure 
"quality 11 (i.e., wages earned and retention) even though this probably 
means numbers served will go down and costs will go up? The strength 
of effect will be relevant in these decisions. For example, it may be 
deemed acceptable to trade off a 15% increase in competitive closures for 
a 9% increase in costs, but not acceptable to increase costs 15% in order 
to achieve a 20% increase in competitive closures. Consideration of 
these kinds of tradeoffs must occur in the process for setting perfor- 
mance objectives at the end of the short-term period. Once the analyses 
are conducted and the policy decisions regarding tradeoffs are made, per- 
formance objectives can be set. Basically, this again means using the 
estimated scores from the chosen model as minimum performance expectations 
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for counselors and field offices. These numerical objectives should be 
accompanied by training of field staff in the technologies determined by 
the analyses as effective in achieving the desired level of performance 
on each goal. In this way, staff will have the tools to achieve at levels 
which are "realistic," and therefore the agency administration's policies 
can only be seen as "reasonable. 11 

With this, the agency will have completed the first full "cycle 11 of 
the process for setting performance expectations on the full range of agency 
goals. This should be a recurring cycle from that point on. Thus, in future 
years, staff should continue to receive performance reports. Simultaneously, 
SSO would continue its analytic activities to further refine the agency's 
understanding of tradeoffs and technologies which succeed in overcoming 
tradeoffs among goals. From the knowledge gained, the agency can revise 
its service practices and review their effectiveness through subsequent 
performance reports. Proven technologies should then be instituted as formal 
policy. The end result should be constant refinement of practice and improved 
productivity on all agency goals. 

We feel that phasing is the best approach the agency can take in that, 
from the start, it is comprehensive (i.e., all goals chosen by agency manage- 
ment are included in the policy statement to staff, and^ are represented in 
recurrent performance reports). Yet the phasing approach allows for an early 
targeting of top priority goals and a gradual introduction of additional 
goals as the agency learns more about the factors influencing performance 
on different goals. Finally, we would expect that the phasing approach, 
in combination with program analysis and staff training, will generate the 
least resistance from field staff as additional performance objectives are 
introduced. 
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III. A RECOMMENDATION FOR SHORT-TERM 



PRIORITIZATION OF AGENCY GOALS 



In the previous section, we presented our recommendation for a phasing 
approach to the process of institutionalizing productivity goals and per- 
formance objectives. Below, we discuss our recommendations for the specific 
performance indicators which might be used to represent top priority goals 
in the short-term for management and performance evaluation purposes. These 
are the indicators which represent the agency's top priority goals, and for 
which specific performance objectives would be established for counselors 
and field offices. 

BPA recommends that HVRSB draw its indicators from the set of data 
elements developed for the revised VR Program Standards Evaluation System. 
The standards are directly tied to the most common VR agency goals. By the 
end of the short-term period, the agency should be able to specify realistic 
performance expectations on each of the indicators selected to represent 
different agency goals. However, for the short-term period, we recommend 
that the agency use the following as its "top priority 11 indicators: 

• the number or clients closed rehabilitated (# 26s); 

• the proportion of clients closed rehabilitated out of all 

clients who are accepted for service (# 26s/ # 26s + # 28s + # 30s); 

• ^the proportion of 26 closures closed into competitive employ- 
ment (# competitively employed/# 26s); 

• the proportion of noncompetitively employed 26 closures 

showing improvement in life status and functional abilities 

/ # noncompetitive 26s showing improvement \ 
\# noncompetitive 26s J 

• total costs per rehabilitation (total expenditures/^ 26s); and 

• percent of severely handicapped clients accepted for service 



We recommend this set of performance indicators in the belief that the 
agency will achieve the greatest level of improvement in overall produc- 
tivity through management emphasis on these measures of performance. The 
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list is short, and therefore should be "manageable 11 in terms of communicating 
priorities to staff and obtaining an effective response from staff in the 
short term. Further, it continues the agency's current emphasis on rehabili- 
tations, while injecting an expanded set of concerns into the agency's over- 
all set of priorities. These recommendations are consistent with our under- 
standing, gleaned from on-site discussions with staff, of those goals 
which are appropriate for designation as "top priority 11 goals. Further 
detailed rationales for including each indicator as top priority are 
as follows: 

• Total number of 26s : This is the most traditional and basic indica- 
tor of VR agency output, well -understood by all staff. It is the simplest 
and most direct measure of "raw output." It answers the qu^ :ion: "Are 
we producing enough of what we're supposed to produce?" 

• Percent of 26s out of all accepted closures : This data element 
measures the extent of "wasted effort" as inferred by the proportion of non- 
rehabilitated closures accepted into the system. Poor performance on this 
indicator suggests several* possible problems with the agency's (or field 
office's or counselor's) "production technology." 

An excessive proportion of nonrehabilitated closures suggests that 
resources have been used unproductively : for each such closure, the agency 
has incurred costs in terms of service costs, counselor time and effort, 
and support costs such as clerical time, data storage (paper files and 
computer space), and supervisory time (e.g., for case reviews). These 
costs would presumably be lower for 30s than for 28s, but even clients 
closed in status 30 may generate service costs during plan development, 
counselor costs resulting from plan development and recordkeeping, and 
support costs. Though costs are incurred for these clients, there is 
no "payoff" for the agency, in the sense that a rehabilitation did not re- 
sult. (This statement reflects the notion that clients closed 28 or 30 
get no "benefit" from participation in Vk. While such clients may, in fact, 
derive some benefit from VR, and there are those who would argue this is the 
case, BPA recommends the agency follow a policy of "no benefit"-- and there- 
fore, no credit to staff —for nonrehabilitated closures.) Finally, in 
addition to unproductive use of resources, a 28 or 30 closure is likely to 
represent a case where clients were inconvenienced and disappointed, more so 
than a 26, when this could possibly have been avoided. 
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• Percent competitively employed : This indicator is related to the 
previous two in that it is essentially a measure of production. It is, 
however, a measure which takes a finer cut at evaluation of production, 
and as such could potentially cause resistance by counselors. Nonetheless, 
BPA feels that this indicator represents VR's highest goal: that of 
assisting clients to obtain gainful (i.e., remunerative) employment. This 
is the essence of vocational rehabilitation. Further arguments for inclu- 
ding this indicator as a top priority are that: 1) competitive employment 
was noted by the agency staff as an important agency goal; and 2) to the 
extent that the agency improves on this indicator, other indicators of per- 
formance should also show improvement. In particular, increases in competi- 
tively employed closures should increase the average closure wages among 

26 closures, and increase the agency's benefit-cost ratio (as a result of 
increased earnings, taxes paid, and reduced welfare). 

Specification of performance objectives on this indicator should always 
presume some proportion of noncompetitive closures. This is consistent with 
VR's broad mission to serve those most in need and to help all clients 
achieve their "maximum potential, 11 and it is consistent with the notion 
that not all eligible clients will be appropriate for paid employment in 
the competitive labor market. However, the agency administration should 
make clear to field staff that noncompetitive employment closures are to 
be considered the "residual" of their competitive employment performance 
objective. Field staff will need to target those clients for whom non- 
competitive employment is the only feasible vocation. 

• Percent of noncompetitive^ employed 26 closures showing improvement 
in life status and functional abilities : This indicator serves two broad 
purposes. First, it is a production" indicator like the previous three 

in that it measures an aspect of agency output for a particular group of 
clients. We have already implied that noncompetitive closures are "less 
valuable" as a VR output than competitive closures. However, because they 
are still a legitimate type of closure, it is important that the agency make 
sure such clients do, in fact, benefit from VR services. The specific data 
items for measuring life status and functional abilities are currently 
being pretested by RSA. When complete, those items can serve as the basis 
for HVRSB's performance assessments. 
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The second purpose for including this indicator is to provide a check 
agianst familiar "numbers game 11 incentives resulting from traditional VR 
evaluation systems based solely on the number of 26s obtained. It is impor- 
tant that the noncompetitive closure statuses not be used as "dumping 
grounds 11 for clients with whom staff were unsuccessful (i .e. , clients who 
would normally be closed 28 or 30). Given the incentives inherent in the 
first two recommended indicators (number and percent of 26s) the temptation 
may be very great to classify someone as a noncompetitively employed 26 
closure rather than as a 28 or 30 closure. While this strategy probably 
can never be totally eliminated, use of this indicator should give managers 
a tool for monitoring its occurrence. 

# Total expenditures per rehabilitation : Thus far, we have discussed 
indicators pertaining to what can be termed production output . With this 
data element, we link production to resource use. Production can thus be 
evaluated within a standardized context of production efficiency . We recom- 
mend including this indicator in the top priority group for two reasons. 
First, given the agency administration's focus on productivity, this data 
element is an obvious choice for early attcntipn by management. 

Second, this indicator is useful for helping answer management questions 
above and beyond the basic unit-cost-per-rehab issue. Specifically, this 
indicator serves as a starting point for agency investigations into caseload 
flow issues. To see this, we recommend that SSO review the decision-support 
analysis model (decision "tree") developed by BPA for investigating excessive 
costs per rehabilitation. 1 Through use of related information items, this 
decision tree will lead SSO and managers to investigate problems in, as 
appropriate, the agency's client acceptance processes, its overall service 
planning and provision mechanisms, or its procedures for placing clients 
and closing their cases. In addition, of course, the decision tree suggests 
ways the agency might analyze its service and administrative cost structures. 
Thus, by including costs-per-rehab as a top priority indicator, management 
not only ensures attention to a key productivity goal, but also simplifies 
the agency's task in pinpointing the most likely causes of productivity 
problems among many complex causes. 



See Berkeley Planning Associates, The Rehabilitation Executive's 
Evaluation System (TREES) : Data-Based Decision Support Using the Program 
Standards System, Data Element 2(ii). 



ERIC 




17 



Two points should be made regarding the construction and reporting of 
this indicator to staff. Specifically, as currently constructed, this 
indicator includes administrative and other nonservice costs within the 
!l total expenditures 11 component. As such, in its current construction, the 
indicator is not perfectly tailored to the circumstances of counselors, who 
may not directly incur administrative costs. For counselors, the intent 
of this indicator could be satisfied by modifying the expenditures component 
to include those items relevant to counselors. Obviously, this would in- 
clude service costs, but could also include the counselor's salary and 
fringe benefit costs and/or other costs identified by management. Such 
modifications presumably would not be necessary for reporting performance 
at the area or district level, since administrative and other costs should 
be directly attributable to those offices. 

The second point is that the agency should understand the distinction 
between the total costs-per-rehab indicator and the related average life- 
of-case services costs indicator. The latter indicator is, in our opinion, 
less powerful as a management tool in that, if calculated for 26 closures 
only, it excludes all administrative costs and all service costs incurred 
for nonacceptances and nonrehabilitants . While such costs may be unavoid- 
able, they are still essentially "nonproductive" costs which the agency 
would want to minimize. The implications of this difference can be seen 
by a hypothetical example. Assume a district office gets 100 rehabilitations 
in a given year, and that the service costs attributable to those closures 
comes to. $25,000. Here, the average life-of-case service cost for 26s is 
$250. However, the office may have incurred a total of $45,000 during the 
period for all operations, including administrative and support staff costs 
and service costs spent for extended evaluation and nonrehabilitants. 
Here, average total expenditures per 26 would be $450. While excessive 
life-of-case costs might alert management to a pattern of overly-generous 
service planning, it- ignores other possible inefficiencies. An additional 
drawback is that life-of-case cost measures are not constrained as to time: 
depending on time-in-status , costs entered into the calculation could have 
been incurred several years previously. Such an indicator obviously is 
less valuable for day-to-day and short-term planning and management than 
indicators constrained to a specific period (as is total expenditures per 
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rehab). These arguments are not meant to imply that life-of-case cost 
indicators are never useful, but rather that they are less useful as a 
first-cut management tool. For these reasons, we recommend HVRSB use total 
expenditures per rehab as its basic production efficiency measure. 

• Percent severely handicapped : This indicator is recommended because 
of the priority emphasis within the agency and at the federal level on 
assisting those most in need of vocational, rehabilitation services. Inclu- 
sion of this indicator in performance reports will serve to highlight the 
administration's focus on these clients. 

SETTING PERFORMANCE OBJECTIVES ON TOP PRIORITY GOAL INDICATORS 

As with the secondary goals, the agency will want to set initial perfor- 
mance objectives on top priority goals at "realistic 11 levels. Perhaps the 
simplest way to do that is to set objectives at some level slightly higher 
than the agency has experienced in the recent previous years: this would 
be an "historic norm 11 approach. However, the agency should be wary of 
accepting historic levels as the objectives for individual offices, especially 
if the levels are well below agency-wide norms. Such a practice could build- 
in mediocrity from the start. The administration should begin by setting 
performance goals for the agency as a whole, and then lf parcel out 11 those 
goals among offices. These goals should be internally consistent. That 
is, the agency goal for total 26 closures should equal (or be less than) 
the sum .of the individual goals set for the different local offices. Where 
indicators involve rates (e.g., the percentage of successful outcomes), the 
initial presumption should be that the agency-wide goal should hold for all 
individual offices unless that office can justify why a lower rate should 
be set. Currently high performing offices may wish to set even higher 
goals for the future, but the agency should be careful not to let such 
offices set unrealistic goals lest the quality of services be reduced in 
the effort to meet the goals, or alternatively, failure be built in. 

The central office should review the performance of each field office 
with its administrator at the end of each fiscal year. It is strongly 
desirable that some central office presence be seen in the first review 
to give emphasis to the agency's new productivity focus. Any review 
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with administrators should conclude with written statements of intended 
actions to improve performance, and should be followed during the next 
six months with at least one follow-up discussion based on monitoring 
the improvement. Without this follow-up, which signals the field that the 
central office is seriously committed to continuing the emphasis, one can- 
not expect changes in behavior. If the follow-up is not done, it would be 
better if the initiative were not taken at all, lest the central office's 
credibility be undermined. 

The performance measured by the standards should be made part of the 
counselor's personnel file, as well as the district administrator's, and 
specifically considered in promotion and merit increases. That is, the 
expectation should be that counselors will at least perform close to the 
norm established in the state by the typical counselor. Performances well 
above the nc~m should receive praise and reward, while performances below 
the norm need major corrective action. District administrators and super- 
visors should be held accountable by the central office for improving the 
performance of counselors below the norm, with specific monitoring under- 
taken of such improvements once problems are detected. « 

Most importantly, the performance of field offices and districts 
should be reviewed in annual budget negotiations and in allocation decis- 
ions. Offices with high performance should be rewarded with more resources. 
Offices with problematic performance should be provided additional resources 
only if they can demonstrate how such resources would be put to a use in a 
way likely to bring performance up to state-wide norms. In times of fiscal 
austerity, low performing offices should be made the most vulnerable to 
staff reductions and transfers. 

SUMMARY 

The recommendations presented here for a possible goal prioritization 
are based on our understanding of the HVRSB administation's priorities, 
and on our opinions as to those aspects of overall VR productivity which 
are most appropriate for monitoring and evaluation. The agency may feel 
that other indicators or additional indicators are also appropriate for 
inclusion within the "top priority" group. However, BPA feels that for the 
short-term, the recommended indicators will provide the agency with a basic, 
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well-rounded picture of its performance in meeting major goals. To briefly 
reiterate our rationale, the set of indicators will help the agency monitor 
productivity in terms of: 

• production output: 

raw output (number of 26s) ; 
"wasted effort" (percent 26s); 

production of "best quality" outcomes (percent competitively 
employed) ; and 

benefit to clients with less than fl best quality" outcomes 
(percent of noncompetitive closures showing improvement 
in life status and functional abilities); 
o production efficiency (total expenditures per 26 closure); and 

• appropriate targeting of services (percent severely disabled). 

The agency could opt to include indicators which more finely measure 
each of these three broad aspects of productivity. For example, production 
output could also be measured with such indicators as average wage of 26s, 
retention of employment, and reduction in welfare benefits. Efficiency 
could be measured in terms of costs per different types of services and 
other detailed cost breakdowns , and targeting could be measured in terms 
of the proportion of clients with different disabilities. For the short 
term (again, two to three years), we recommend that the agency not expand 
the group of top priority indicators. BPA feels that HVRSB's long-range 
productivity objectives will be best served by focusing on a few basic 
indicators and building up to an expanded set. This is not to say that, 
even in the short term, the agency would make no use of additional data 
items. In fac.t, extensive use should be made of them. We have already 
discussed how indicators representing "other 11 goals would ideally appear 
in the recurring performance reports. These might include such outcome 
and productivity measures as the proportion of clients earning the minimum 
wage; the proportion of clients retaining employment over time; average 
change in clients 1 earnings after VR; the agency's benefit-cost ratio; and 
other data items suggested in BPA 1 s revised Performance Standards System. 

But, again, in order to avoid diluting the agency's initial efforts 
at improving productivity, it is recommended that for the short term these 
indicators be used for information only, rather than as items on which 
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performance will be evaluated. Other data items, such as detailed cost 
and caseload breakdowns, should be considered as management tools for 
analyzing performance and informing management decision -making, rather 
than as productivity indicators. Take the example of time-in-status 
statistics for monitoring case flow. We would argue that it is important 
to know this information, not for the purpose of punishing counselors with 
slow-moving caseloads, but rather for the purpose of pointing out to the 
counselor that slow case movement now can lead to caseload bottlenecks and 
low productivity later. It is important for the agency to understand the 
distinction between an "end-state" productivity measure appropriate for 
use in staff evaluation, and an "in process 11 indicator useful as a manage- 
ment tool. 
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IV. AN EXAMPLE OF THE USE OF PROGRAM 
INFORMATION IN MANAGING VR OPERATIONS 

The purpose of this section is to provide the agency with a brief, 
hypothetical example of the reports and analyses the agency might use in 
investigating performance problems of a particular nature. The example 
is presented in terms of the performance of individual field offices around 
the state. 

The first step in using information for management is to obtain basic, 
descriptive data on the performance of field offices. With this, possible 
performance problems will become apparent. Table 1 shows one possible 
format for such a report. BPA recommends that the agency develop such 
a report for distribution to the agency director, the chief of SSO, and 
each field office administrator. (The cell entries of this table denote 
those cases where field office performance is substantially below the 
office's performance expectation on the indicator. These entries are for 
illustration and discussion. An actual performance report would have 
actual numbers denoting the performance obtained by the office.) This 
table format is useful in several ways. In one summary format, information 
is presented which allows for an assessment across field offices in terms 
of a particular aspect of performance (i.e., reading down the "indicator 11 
columns), and which allows for an assessment within a field office across 
all aspects of performance (i.e., reading across the "field office 11 rows). 
In addition, readers can compare the performance of a given field office 
to the aggregate performance of all VR services field offices ("Total, VR 
Services"); to the VR services section of Blind Services ( tr Totai, Blind 
Services (VR)"); and to the combined performance of all VR services 
("Agency Total") . An analogous report could be prepared for distribution 
to counselors, supervisors, and field office administrators. The columns 
of such a report would be the same, but the row entries would consist of 
individual counselors, the aggregate field office figures, and £.»>vr/es for 
VR and Blind Services and the agency total. 

The cell entries in Table 1 have been chosen to illustrate a variety 
of possible performance problems. The configuration of "problems" has been 
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Comparative Performance on Top Priority Indicators Across Field Office, Service Branch, and Agency 
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set out for discussion only. The agency actually may never see certain 
of the situations in its own monitoring of performance. A table such as 
this provides the agency with a tool for interpreting performance on individual 
indicators, within the context of the agency's overall productivity evaluation 
system. Indirectly, summary tables such as these can suggest the types and 
levels of tradeoffs involved in striving to meet conflicting goals. 

The lack of any notations in the 'Total, VR Services," 'Total Blind 
Services (VR)," and "Agency Total" rows indicates that, overall, the service 
divisions and the agency are meeting all performance goals . As such, the 
problems noted across field offices cannot be attributed (at first glance, 
at least) to state-wide conditions. Were this the case, the agency would 
want to look to those field offices which had met goals to determine factors 
which allowed the offices to succeed, and to possibly use those offices as 
models for the rest of the agency. As it is, were Table 1 an actual perfor- 
mance report, the assumption could be made that the offices' various "problems' 
result from circumstances within the particular field office. The table 
suggests the following problems: 

• Office A has met its output and efficiency goals, but has failed 
to admit a sufficient proportion of severely handicapped clients. This may 
in part explain why the office has been able to meet its other productivity 
gojils. For this office, the agency may want to investigate the decision 
rules used by counselors and intake staff: Are acceptance criteria too 
restrictive? Alternatively, diagnostic services and extended evaluation 
might be expanded to assist counselors in assessing client potential and 
in writing service plans. Other reasons may appear on investigation as 
to why the office serves too few severely handicapped. Agency management 
should probably expect some drop in performance on the remaining produc- 
tivity measures as this problem is corrected. 

• Office B has met all its objectives except that for percent com- 
petitively employed. Note that, since the office achieved its desired 
number of 26s, and a sufficient proportion of noncompetitive 26s showed 
improvement, the office at least succeeded in benefitting clients for 
whom the "highest goal" was infeasible. Nonetheless, quite possibly 
the office may have closed many clients into noncompetitive employment 
when their original plan called for a competitive employment closure. 
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• Office C is doing well in meeting the major output, efficiency, and 
targeting goals. However, an unacceptably large proportion of its non- 
competitive employment closures show no improvement. This immediately sug- 
gests one of two things: either the office has not learned how to effectively 
help clients with noncompetitive employment plans, or some "dumping" of 
cli_ents_whp_ should have been closed 28 or 30 has occurred. In the latter 
case, if the "truth be known" (i.e., counselors coded closures "by the book"), 
we might expect the office's performance on number and percent of 26s to 
drop. Fortunately, the percentage of competitively employed closures would 
not be affected. 

• Office D has sufficient 26 closures and those closures meet goals 
for competitive employment and improvement. However, too many clients were 
closed unsuccessfully. This is a case where the office may have overworked 
itself admitting clients and simply could not succeed with the added load 

of clients over and above their objective for number of 26s. However, since 
costs per 26 are acceptable, it does not appear that large amounts or resources 
were expended for "nonproductive" uses. Still, effort has been "wasted," 
and the goal is to see if the office can more productively target the clients 
it accepts and the services it provides. 

• Office E failed to rehabilitate enough clients, though of those 
rehabilitated, adequate proportions obtained competitive employment and 
showed improvement. Further, there was no "wasted effort," inasmuch as a 
sufficient proportion of all closures were rehabilitated. However, even 
this is problematic, given the following logic: If the number of 26s is 
too small but the proportion is acceptable, then the number of 28s and 30s 
is "too small." Stated differently, the total number of closures is too 
small. (This is reflected in the poor showing on costs per 26.) The office's 
task is to see if it can increase the total number of closures, thereby 
hopefully increasing the number of 26s in the process. More clients 

must be pushed through the system faster. Hopefully, this can be done 
while maintaining closure quality (i.e., by maintaining objectives for 
competitive employment and improvement). Here, SSO would want to analyze 
whether resources could be used more efficiently; for example, through 
lower cost and shorter duration plans (which may have negative implications 
for closure quality), or through admitting different types of clients (which 
may have negative implications for the agency's targeting goal). 
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• Office F has a situation where both the number and proportion of 
clients closed 26 are too low. In the absence of actual data, we cannot 
know if the total number of closures is also too low (as with Office E) . 
If so, this office needs to push more clients through faster, like Office E. 
If enough total closures are occurring, then the office needs to better 
target its clients and do a better job in providing service (as was the 
case with Office D) . 

Having identified the broad aspects of productivity on which different 
offices perform poorly, the task becomes one of investigating the reasons 
for inadequate performance. Then, of course, findings from these investi- 
gations should be transformed into action plans intended to improve perfor- 
mance. We present a specific, hypothetical example of what SSO might find 
in investigating a performance problem, and how SSO would go about the 
task. We use Office D from Table 1 above for illustration. Following 
this, we discuss the general issues SSO might address in conducting 
decision-support analyses. 

Recall that Office D failed to rehabilitate a sufficient proportion 
of clients, but that the goal for number of 26s was met. In the course of 
analysis, the following findings might result. SSO might obtain data on 
closure status by disability, as in Table 2. As Table 2 suggests, the 
office was at least successful with deaf clients. Immediately one 
decision response is possible: restrict entry by deaf people. However, 

this is probably an unattractive option, and so the task becomes one of 
determining more effective approaches to succeeding with deaf clients. 
Probably the first place to look for an answer is to observe the pattern 
of services received by deaf clients and to see which among the services 
appears most related to successful rehabilitation. Table 3 presents the 
hypothetical results of such an analysis, in which two services — on-the- 
job training and facility services — appear most successful. (Such 
findings might be corroborated by obtaining similar cross-tab data for 
other offices.) While the management response may seem obvious -- make 
sure these services are given to deaf clients -- in fact, SSO's investiga- 
tive task is not yet complete. First, with respect to facilities services, 
it is important to know which among the possible facilities are most 
successful with deaf clients. The hypothetical findings in Table 4 suggest 
that facilities A and C are most successful. SSO would want to find out 
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Table 2 
Office D 

Closure Status by Disability (Hypothetical) 
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Total 
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Table 3 
Office D 
Deaf Disability Clients : 
Closure Status by Services Received (Hypothetical) 
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Table 4 
Office D 

Deaf Clients Receiving Facility Services : 
Closure Status by Facility (Hypothetical) 
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why. Is it because those facilities provide specific types of service? 
Do they tend to deal with younger or older clients? Do they have linkages 
with potential employers? It may be that these facilities' successes are 
simply artifacts of such factors, in which case it may not be warranted to 
send all deaf clients to those facilities. Finally, even if warranted on 
the basis of success rate, SSO will need to consider the cost implications 
of sending an entire group of clients through facility service. 

With respect to on-the-job training, other questions are relevant. 
Can the office develop enough OJT slots to accommodate all deaf clients? 
Are all deaf clients sufficiently prepared to enter OJT slots or will 
extensive training, appliance, and maintenance payments be .required to 
bring these clients to the necessary level of readiness for OJT? Only if 
these and other issues can be resolved is it appropriate to target OJT 
slots to deaf clients. (Again, this presumes that such targeting will 
not effectively exclude nondeaf clients from OJT.) 

In this brief example, we have tried to show the kinds of questions 
and analyses which would be relevant in evaluating and attempting to improve 
productivity. In general, SSO and service staff will want to ask questions 
regarding such issues as: 

• client selection : case difficulty and client motivation 
are obvious contributing factors in successful outcomes; 

• service effectiveness: outcomes will be affected by the 

use of diagnostics and extended evaluation to assess potential, 
targeting of services to maximize success, and existence and 
effectiveness of placement; 

• service p rocess problems : cases may be allowed to languish 
in certain statuses, client-counselor interaction may be 
poor, etc . ; 

• service planning: job goals may be inappropriate for the 
labor market, or may be beyond the clients' capacities; and 

• inefficiencies: overly expensive plans may be the norm, 
when less expensive plans could suffice; high cost vendors 
may be used too often. 
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HVRSB will want to encourage a "partnership 11 between SSO and field 
managers. Specifically, managers must clearly understand that the 
agency's new productivity objectives rise out of the information goal 
assessment and prioritization activities. They must understand how 
information can be used, as discussed here and in other VR-TECHS. 
Finally, they must be able to call upon SSO to assist them in identifying 
causes of problems and suggesting solutions. SSO must be prepared to 
respond to inquiries from the field and to apply their analytic creativity 
to ferret out the true causes as completely as possible. In addition, 
SSO must fully understand the capabilities of HVRSB 1 s new MIS, and be 
prepared to remediate gaps in the system when necessary. This partner- 
ship ideally will be further expanded to involve the local field staff 
in thinking through the causes of performance and how to improve performance. 
After specific problems are identified, the field staff should draw 
counselors into the discussion. This will provide a major training 
experience for counselors which can only improve the agency's internal 
capabilities. Again, the quality circle activities, with or without SSO, 
would be an ideal vehicle for encouraging thinking about performance in 
this way. 
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